Modeling object interactions and facial expressions

ABSTRACT

A method of using parametrical representations in modeling and animation is disclosed. This method is built directly on the inherent properties of parametric curves. Parametric curves are constructed and associated with geometric models, while at the same time the properties are used to represent the underlying mechanism of deformation. This method includes the steps of building parametrical representations on a geometric model, associating vertices of the model with different ones of the parametrical representations to effect movement thereof, and animating movement of the vertices to simulate relative motion therebetween.  
     The invention discloses a method in which simulation of object interaction and object deformation can be integrated into a unified framework through skeletal trees. The method preprocesses a graphic object and segments the object into a number of contexts at surface slope and curvature discontinuities. Then, control points are generated to represent these contexts. Based on these control points, a skeletal tree of the object can be built. Assisted by spatial decomposition, the skeletal trees can be directly used in the simulation of object interaction.

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of PCT/AU01/01539, filed Nov.27, 2001 in the Australian Patent Office.

FEDERALLY SPONSORED RESEARCH

[0002] Not applicable

SEQUENCE LISTING OR PROGRAM

[0003] Not applicable

[0004] The present invention relates to the fields of scientificvisualization, computer graphics, virtual reality and simulation androbotics and to the fields of realistic modeling and animation, morespecifically human facial expressions.

BACKGROUND TO THE INVENTION

[0005] With the increase of computer power and widespread use ofcomputer graphics, scientific visualization is facing the challenge ofmodeling physical objects and their interactions, not only realisticallybut also in a real time rate. To achieve realistic simulation,volumetric representation has advantages over traditional surfacerepresentation, primarily because volumetric data is a more naturalrepresentation for modeling of objects and is able to visualise theinternal structure. However, volumetric data is massive and lacks asense of connectivity.

[0006] It has been proposed to have a linked volume representation formodeling volumetric object interactions, in which each element in asampled volume is explicitly linked to its six nearest neighbors. Bymodifying these links, this method can simulate various objectinteractions, such as collision detection and response, objectdeformation, carving, cutting, tearing, joining, etc. The method can beconsidered as an improved version of the original occupancy map method.This method could consume a vast amount of memory and exhibit inertialand damping behavior. A probability model has also been proposed whichcan be considered as an extension of the surface-based approach with theaddition of means taking account of the lack of explicit surfaces.However, most real-life objects have well-defined surfaces. Foramorphous objects, such as fog, clouds and smoke, the well developedtool of fry sets can probably be used in the simulation of objectinteraction. Thus interaction analysis can be decoupled from specificrepresentations.

[0007] Articulated skeletons are used as a tool for animation and it hasbeen proposed to construct skeletal trees using distance transform. Themethod is lossy in terms of reconstruction and cannot properly solve theproblem of object interaction. A voxel-coding based skeletalizationmethod has been proposed but does not satisfy real time requirement Onthe other hand, the problem of collision detection has been extensivelystudied from different perspectives. Most of the previous work howeverfocused on surface-based objects. Their approaches usually requiretime-consuming preprocessing, they treat collision detection andresponse as distinctly different operations, and can not processnon-convex objects well.

[0008] The present invention presents a novel method that aims to solvethe various problems in an integrated approach. The preferred methodpreprocesses objects and segments them into a number of contexts atsurface slope and curvature discontinuities. Then rule-based fuzzy logicis exploited to generate control points to represent contexts. Askeletal tree can be built using these control points. Then exploitingconcurrent computing, the skeletal tree can facilitate animation andsimulation of object interactions. The preferred method work ismotivated by the fact that a tree structure is a natural and economicalrepresentation of objects, also the combination of the representation,and concurrent computing can enable accurate and timely simulation ofobject interactions, at a low resource cost. The present method makes noassumption about object shapes, convex or non-convex, moving directions,collision positions.

[0009] The preferred method can accurately detect object collision, andin some sense, integrate collision detection and response. Furthermore,the entire process remains within the volume domain and does not need toconstruct hierarchial bounding volumes, which may be time-consuming. Thegenerated tree structures can also fit well with the animation pipeline.

[0010] Human face modeling and animation can be applied in such diverseareas as low bandwidth teleconferencing, man-machine interface,entertainment, surgical facial planning, etc. There are five basic kindsof known methods for facial modeling and animation in the art:interpolation, parameterization, muscle-based, physics-based, andperformance-based.

[0011] Interpolation resembles the conventional key-frame approach. Itspecifies desired expressions at particular points of time and generatesintermediate frames using the in-between algorithms. This approach isvery labor and data-intensive.

[0012] Parameterization uses parameterized surface patches toapproximate human faces and utilizes Free-form Deformation (FFDs) foranimation. This approach is one of the most widely used techniques,however, parameterization lacks an accurate anatomical basis.

[0013] Muscle-based approaches simulate various muscle movements usingsimplified mathematical models. For example, “muscle vectors” have beendeveloped to approximate linear, sphincter and sheet muscles.

[0014] Physically based methods using Newtonian physics aim to simulatethe underlying mechanics of muscle movements. Drawbacks of this approachinclude that it needs high power graphics workstations and real-timeperformance is not achieved. In terms of dynamic modeling of com[plexsystems, it may be worth noting some other well developed areasa such ascontrol engineering, in which there is a paradigm shift from thepredominance of plant modeling to the inclusion of more intuitiveapproaches as fuzzing rules.

[0015] Performance-based animation uses data from real human actions todrive the virtual character. This approach uses motion tracking devicesand additional data processing.

[0016] A method of the present invention describes a new NURBS-basedfacial modeling and animation method. Generally, in the real commercialworld, facial animation is still done using trial-and-error methods andis very time consuming.

[0017] FACS (Facial Action Coding System) is widely used in the codingof facial movements. FACS breaks down any facial expressions into someprimary muscle action units. These discrete facial units can act asbuilding blocks to parameterize facial muscle movements. However, FACSis not ideal for animators, since it is not an easy task to break downan arbitrary expression into-the action units.

[0018] FFD (Free Form Deformation) is extensively used as a low-leveldeformation tool in facial animation. FFD embeds an object in a spaceand deforms the space, thus causing the object to become deformed. Theconventional FFD has some disadvantages for facial animation since it isreally a global deformation method and not suitable for complex, subtleand local deformations. Attempts have been made to use more specific FFDtechniques in facial animation.

[0019] The preferred embodiments of the present invention provides anovel method in modeling and animating human faces. NURBS curves arebuilt to correspond with facial muscles, and the vertices on the facialmodel are divided into different units and then associated with theNURBS curves geometrically. To more realistically simulate theanatomical nature of muscles, fuzzy logic is exploited to assign anyvertex a membership to a facial unit. By modifying the NURBS curves,i.e., by modifying the weights of their control points, variousrealistic facial expressions can be simulated. The NURBS curve, which isa specific form of parameterization representation, is used as anunderlying control mechanism in this method. The present inventionrepresents a new paradigm in quantative simulation and animation and isdramatically different from the traditional parameterization methods inwhich parameterization representations are used to approximate the humansurface.

[0020] A NURBS curve is defined by its control points, weights andknots; modifying any of these changes the shape of the curve. Thepresent method simulates the facial muscles by changing the weight ofthe NURBS curves, which is geometrically clear and intuitive. The methodof the preferred embodiments allows any vertex on the facial model to becontrolled by a number of NURBS curves which is anatomically analogical.This can facilitate easier control of the high-priority facial features,such as eyes, mouths, etc., and allow more realistic animation. Thismany-to-one mechanism has not been systematically exploited bypreviously proposed facial animation methods, such as muscle-basedmethods and various FFD methods.

[0021] Modeling facial movements using NURBS curves is a geometricalmodeling method. The NURBS curves are used to control and coordinate themovement of the vertices. This approach can possibly be combined withmuscle-based and physically based methods. This method can alsopotentially be combined with performance-based methods by usingtechniques for curve fitting Another logical extension of weightmodification is to reposition the control points of the NURBS model.

OBJECT OF THE INVENTION

[0022] It is an object of the present invention to provide a method ofmodeling object i9nteractions and facial expressions which substantiallyovercomes the above mentioned disadvantages. At the very least theobject of the present invention is to provide an alternative to knownmethods.

DISCLOSURE OF THE INVENTION

[0023] According to one aspect of the present invention there isdisclosed a method of context based skeletal tree generation for use inanimation and simulation of object interactions, said method comprisingthe steps of preprocessing graphic objects, segmenting processed graphicobjects into a plurality of contexts at surface slope and curvaturediscontinuities, generating control points to represent said contexts,building a volumetric skeletal representation from said control points,and wherein said volumetric skeletal representation is manipulated tofacilitate animation and simulation of object interactions.

[0024] Preferably, the volumetric skeletal representation is in one forma skeletal tree and the means to generate the skeletal tree includesboth bottom-up and top-down.

[0025] Preferably, rule based fuzzy logic is used to generate saidcontrol points and the step of segmenting preprocessed graphic objectsinto a plurality of contexts, includes edge detection, edge tracking andcurvature discontinuity detection. The edge detection is preferablyaccomplished by using gradient or Laplacian operators, sthe edgetracking by using omnidirectional tracking, and curvature discontinuitydetection is accomplished by measuring the k-slope and k-curvature andcomparing them to predetermined threshold values. Furthermore, the stepsof providing data structures representing contexts of the segmentingprocess, said data structures having a queue containing x-y co-ordinatesand/or chain code values of context points and pointers to neighbouringcontexts. Preferably the preferred forms include the steps of applyingcontext dimension estimation and/or resegmentation which is accomplishedby scanning the context both horizontally and vertically, constructing arun-length histogram which gives the occurrence of length of differentruns within the context, the dimension of the context being therun-length corresponding to the maximum value of the histogram. Thepreferred method further includes the steps of comparing dimensions ofdifferent contexts and selecting a standard dimension, and using thestandard dimension as a parameter to re-segment.

[0026] Preferably, the step of applying fuzzy logic includes the step ofproviding a quadtree to represent the object for space efficiency.

[0027] Preferably, there are two types of threads, one type runs at ageneral level, which roughly tests if objects will collide, the othertype runs at a detailed level, which accurately and in time checks theinteractions of objects and a general level thread can spawn a detailedlevel thread whenever necessary. If multiple objects are involved, thescheduling scheme of multiple objects can be built on the top, whereinthe objects are individually modeled in their local coordinate systemand skeletal trees are set up for them respectively, and in the courseof their movement, their control points are queried by a general levelthread to check if there will be a collision. The general level threadmaps variables in individual local coordinate systems to a worldcoordinate system, and take account of object orientation, relativemoving directions, a search is initialized from the root of a skeletaltree, towards those control points that are closer to another object inspace, and if the general level thread conservatively determines, thatthe contexts where one pair of control points are situated, may collideat some time, then a new detailed level thread can be spawned to carryout accurate detection, and pass the pair of control points to thisdetailed level thread. The hierarchical depth in the skeletal tree wherea general level thread spawns a detailed level thread can vary, anddepending on the objects and the task-sharing requirements, it can be auser defined control point, or a context level control point, or asub-dividion level control point.

[0028] For context level tree nodes, the triggering condition fordetailed level threads can be that, if the distance between the controlpoints is less than 1.5 times the sum of the context dimensions, whichhave been calculated during the segmentation process.

[0029] The detailed level thread can use either a passive or an activemethod to detect collision, in a passive method, for each skeletal treenode in skeletal subtrees where control points are situated, from itscoordinates, the quadtree or octree node which contains the skeletaltree node can be found whereby the passive method then translates thepositions of terminal quadtree nodes branched from the nodes justdetermined to a world coordinate system and check if there is anyoverlay or collision, if the terminal nodes of quadtrees or octrees,which are being checked, have only been subdivided into largerdimensions (length, height and width), they can be further subdivided bythe detail level thread at run time, wherein in an active method, wavefronts resembling context surfaces are pre-sent along the objects'movement directions, again in a world coordinate system, thus predictingcontact point and time, the wave fronts are constructed as parametricsurfaces if available, or they can use the node comparison approach asin the passive method.

[0030] For collision response, the skeletal tree can be convenientlyused in the analytical simulation and the animation afterwards, aftercalculating for deformation, the coordinates associated with therelevant nodes in the skeletal tree will be updated. Then, the skeletaltree can be used to guide the update of the quadtree (octree)representing objects, and the movement of nearby elements, such as othertree nodes of the quadtree (octree), can be interpolated. If an objectis properly segmented and subdivided, and the nodes in the skeletal treerepresent the object in sufficient details, the updated quadtree(octree) will faithfully represent the object after the update. Theevent driven methods are used in general level and detailed levelthreads to speed up processing wherein usage of multithreads is toincrease responsiveness and efficiently use system resources forinteraction modeling without conflicting with the main tasks.

[0031] According to another aspect, a method is disclosed to use avolumetric skeletal representation representing an object forinteraction detection and response, wherein nodes are searched fordetection and position of nodes are updated for response, saidrepresentations being generated either bottom-up or top-down or both.

[0032] Preferably, the skeletal representations include a tree and thenodes includes treenodes, and the method to generate the skeletal treeincludes context-based approach, fuzzy rule approach, automation inprocessing. The method preferably includes the use of trees intree/concurrent computing in detecting interactions includesmultithreads, wherein collision locations and time are predicted and theskeletal trees are updated in interaction in response for continuousstimulation and wherein the method includes an active method whichdynamically generates the tree in greater detail in prospectivelocations of the collision.

[0033] According to another aspect of the present invention there isdisclosed a method of using parametrical representation to control andrepresent surface deformation to simulate movement in animation, saidmethod including the steps of building parametrical representations torepresent vertices in the topography of the surface, associating saidvertices with different ones of the parametrical representations toeffect movement thereof, and animating movement of the vertices tosimulate relative motion therebetween.

[0034] Preferably, the method is used to animate facial expression,whereby the vertices in the topography of the surface are assigned tovarious portions of facial anatomy including muscles and wherein theparametrical representations are weighted to differentiate the differentportions of the facial anatomy. The weighting parameter is preferablychanged to simulate facial expressions.

[0035] Preferably, the parametrical representations are NURBS curves andwherein position of the NURBS curves are predetermined by the locationwhere simulated muscles are attached to facial bone, each NURBS curvehaving at least one control point.

[0036] Preferably, fuzzy logic is used to determine the association ofvertices with NURBS curves and the weights of the NURBS curves arevariable to modify the NURBS curves.

[0037] Preferably, the parametrical represntations and/or their changesare sent to a remote location via a communication link to generatedeformation at remote location.

BRIEF DESCRIPTION OF THE DRAWINGS

[0038]FIG. 1 a-b show general flowcharts of context-based skeletal treegeneration and its application in modeling object interactions.

[0039]FIG. 2 shows a general procedure of context-based segmentation.

[0040]FIG. 3 shows a general procedure of skeletal tree generation.

[0041]FIG. 4 shows examples for generating control points usingrule-based fuzzy logic.

[0042]FIG. 5 represents a simple example to show detection of objectinteraction.

[0043]FIG. 6 shows class diagram that illustrates modeling objectinteraction.

[0044]FIG. 7 shows class graphobj in FIG. 6.

[0045]FIG. 8 shows class trajectory in FIG. 6.

[0046]FIG. 9 shows class generalthread in FIG. 6.

[0047]FIG. 10 shows class detailthread in FIG. 6.

[0048]FIG. 11 shows class space in FIG. 6.

[0049]FIG. 12 shows approximate locations of control polygons of NURBScurves.

[0050]FIG. 13 shows positioning control points of NURBS controlpolygons.

[0051]FIG. 14 shows separation of facial units.

[0052]FIG. 15 shows shows the determination of a gradient for a vertexaround the mouth.

[0053]FIG. 16 shows association of a vertex with a NURBS curve.

[0054]FIG. 17 shows an example of determining membership, to theforehead and one eye for a vertex on the area

[0055]FIG. 18 shows an alternative calculation of ration to curve samplepoint.

[0056]FIG. 19 shows vector analysis combined with physical basedsimulation

BEST MODE OF CARRYING OUT THE INVENTION

[0057] The details of the preferred embodiments of the present inventionwill be discussed in three sections: context-based segmentation,skeletal tree generation, and simulation of object interactions, whichalso represent operations in a pipeline fashion. FIGS. 1a-b showsgeneral flowcharts for the operations described in this specification.The discussion in this specification mostly focuses on simple 2D objectswithout holes. However, the basic principle also applies to complicated3D objects with holes.

[0058] A context in this specification means a surface segment thatexhibits a high level of uniformity, that is, a continuous surface patch(C⁰ continuity) with a gradually changing tangent (C¹ continuity). In hephysical world, a context is usually an integrated part and can becontrolled as a whole. Context-based segmentation has been used in somecomputer graphics applications, such as context-sensitive normalestimation.

[0059]FIG. 2 shows the general procedure of context-based segmentationfor the present invention. The segmentation operation in FIG. 2 includesthree steps: edge detection, edge tracking and curvature discontinuitydetection. In 2D image processing, there are numerous methods toimplement these three operations. For example, edge detection can beaccomplished by using difference operators, such as gradient orLaplacian. Omnidirectional tracking can be used for edge tracking.Curvature discontinuity can be detected by measuring the k-curvature(C¹) and change in k-slope (C⁰) while an edge is being tracked.Discontinuities can be the locations where both the k-curvature andchange in the k-slope are larger than some threshold values. Thethreshold values and k are application dependent. It has been found thatk can be set to 5˜10 for images of various sizes. The absolute value forthreshold of k-curvature (C¹) can be set to 10 degrees; the absolutevalue for threshold of the change in k-slope (C⁰) between neighboringpixels can be set to 20 degrees. In the following, the threshold valuesare all set for absolute values.

[0060] After these steps, the surface of a 2D object will be partitionedinto different contexts at discontinuities. The partition is based onlocal criteria, i.e. local slope and curvature. Other local criteria caninclude distances between neighboring discontinuities. For example, ifthe distance between a discontinuity and its left discontinuity isalmost the same as that between its right discontinuity; and itscurvature and slope calculated on these discontinuities are smaller thansome threshold values, e.g. which can be set as 30 and 45 degreesrespectively, then the discontinuity can be considered as not meetingstronger local criteria.

[0061] The outputs of the segmentation operation can have datastructures representing the edge and discontinuity locations. The datastructure for the edge can be a linked list containing x-y coordinatesof edge points for a 2D object (x,y,z coordinates for 3D objects). Thedata structure representing discontinuities can be a linked list whichholds the indices of the discontinuities to the first linked list.

[0062] After segmentation, an animator can decide whether to apply thenext two operations: context dimension estimation and re-segmentation,and whether to apply them to all or some of the contexts. The purpose ofre-segmentation and sub-division within contexts is to provide abalanced and fine grained structure to other operations in theprocessing pipeline, eg skeletal tree generation and interactiondetection.

[0063] The dimension of a context can be the distance between its twoendpoints. Alternatively, to estimate the dimension or size of acontext, the context can be scanned horizontally and vertically. Then, arun-length histogram, which gives the occurrence of the length ofdifferent runs within the context, is constructed. The dimension of thecontext can be the run length corresponding to the maximum value of thehistogram.

[0064] Re-segmentation is based on global criteria, e.g. a globallyselected standard dimension. During the re-segmentation operation,first, dimensions of different contexts are compared and a standarddimension is chosen. Then the standard dimension is used as a parameterto re-segment contexts, e.g. contexts whose dimensions are much smallerthan the standard dimension are merged into neighboring contexts, andcontexts whose dimensions are much bigger than the standard dimensionare re-segmented into a number of contexts. A context whose endpointshave a curvature and a change in slope larger than some thresholdvalues, e.g. which can be both set as 30 degrees, will not be mergedinto its neighboring context.

[0065] Subdivision within contexts is a bit like re-segmentation; acontext is further divided according to certain criteria. These criteriacan be a certain distance from, or an average rotation relative tocontext endpoints or neighboring subdivision locations, or a certainnumber of subdivisions within the context. A context whose dimension issmaller than a global threshold may not be subdivided.

[0066] The final output of context-based segmentation can include datastructures representing subdivided contexts, e.g. a linked list whoseelements are arrays and each array holds indices of context endpointsand subdivision locations of one context. Another benefit ofcontext-based segmentation is the surface normal can be representedeconomically. Since within one context, the surface normal only changesgradually. The surface normal is useful for simulation of physicalinteraction, such as haptic interfaces.

[0067] For objects with holes, the external and inner contours canseparately go through the operations in FIG. 2. Then by connectingbetween context level segmentation locations of external and innercontours, an object with holes will be partitioned into a number ofpatches, which can be considered as contexts. For a 3D object, 2D slicesfor all three orientations namely X, Y, Z, can be examined.

[0068] Also, the labeling of discontinuity voxels can be achieved as aby-product of voxelizations algorithms

[0069] For volumetric objects that lack explicit surfaces, the objectscan be preprocessed to construct surface representations.

[0070] In a mathematical definition, the skeleton of a shape is thelocus of the symmetric points of the local symmetries of the shape. Theskeletal tree used in this specification does not fit strictly withinthis mathematical definition; the skeletal tree is more closely relatedto the skeletal structure used in computer animation. The aim ofutilizing a skeletal tree in the present invention is to facilitate fastanimation, interaction simulation, such as collision detection andresponse. Generation of control points for the skeletal tree can havetwo phases. At the first phase, fuzzy logic and rule-based reasoning isutilized to generate control points at low levels of the skeletal tree,making these control points representative for contexts or subdivisionswithin related contexts. At the second phase, control points at the highlevels of the skeletal tree can be positioned interactively by ananimator, or using skeletonization methods on the basis of simplifiedobject representation, such as object discontinuities.

[0071] Fuzzy systems combine one or more input variables, defined interms of fuzzy sets, with a collection of fuzzy rules to produce anoutput value. Designing the parameters for a fuzzy system usuallyinvolves:

[0072] 1. Determining appropriate inputs and outputs;

[0073] 2. Specifying fuzzy sets to be associated with each variable;

[0074] 3. Deciding fuzzy rules;

[0075] 4. Specifying the shape of membership functions.

[0076] As an application of fuzzy systems in this specification, theinputs can include any of the following: locations of context endpoints,subdivision locations, control points already obtained and datastructures representing the contexts; the outputs are the locations ofcontrol points. Fuzzy sets and the shape of membership will be similarto those in other fuzzy logic applications, e.g. fuzzy control.Therefore the following will emphasize the fuzzy rules used indetermining control points.

[0077]FIG. 3 shows the general procedure of the skeletal treegeneration. FIG. 4 shows examples of step 1 and step 2 in FIG. 3: togenerate a control point for a subdivision within a context by usingfuzzy logic, as well as to generate a control point for a context.

[0078] In FIG. 4(a), suppose the convex surface between A and B has beensegmented as one context using the method discussed in the previousdescription, and the context is further subdivided at E and F. Connect Aand E, the middle point of AE is u. In FIG. 4(a), context AB is convex,we can define the distance d between u and curve AE (surface for 3D) aspositive. In FIG. 4(b), since u is outside the non-convex object, we candefine d as negative. We can further define the distance between u andthe final control point v as c, and the sign of c indicates if c and dlie on the same side of the line segment AE (positive, as at the sameside). The following fuzzy rules can be used to determine the locationof the control point for subdivision AE:

[0079] a If d is poslarge, then c is negsmall.

[0080] b. If d is possmall, then c is neglarge.

[0081] c. If d is zero, then c is large within the object.

[0082] d. If dis negsmall, then c is poslarge (+|d|).

[0083] e. If d is neglarge, then c is possmall (+|d|).

[0084] In rule a)˜e), large or small is relative to the distance Au orAE.

[0085] The determination of a context level control point CP can alsomake use of fuzzy logic, which is similar to generating a control pointfor a subdivision, but based on context endpoints (A and B) and otheredge points of this context. For non-convex context, CP can be mirroredinside the object.

[0086] In FIG. 4, distances among geometrical elements are used toimplement fuzzy logic reasoning; it is obvious that the same principlecan be implemented by making use of geometrical angles. Connectionbetween context level control points and those of related subdivisions,as shown in FIG. 4(c), will generate subtrees of the skeletal tree.These subtrees are at the lowest level of the skeletal tree, andprobably isolated from each other. All nodes in these subtrees should beassociated with coordinates of the control points. The purpose of thisassociation is to speed up collision detection as discussed in thefollowing description.

[0087] The control points generated above, in some sense, will becentral to the context, but may not be central globally. For example, acontrol point for one context may be too close to the surface of anothercontext or out of the object boundary. If subdivision is properly done,then, this problem can only occur with context level control points. Thepurpose of context level control points merging and moving (as in FIG.3) is to solve this problem. These operations can be manuallyaccomplished by an animator or by using computational intelligencetechniques, e.g. fuzzy logic or neural networks. For example, a fuzzyrule can be set for control point merging as: If two context levelcontrol points are too close and their connection doesn't cross aboundary, they will merge into one, taking account of their weights.Another fuzzy rule for control points moving can be set as: If a controlpoint is too close to an object boundary, it will be moved inwards somedistance (maintaining centeredness within the context and globally).After merging and moving, the connections between the control points ofthe contexts and those of related subdivisions should be retained andtheir coordinates updated.

[0088] Now, the animator can manually set some control points at highlevels of the skeletal tree, or adjust some control points that havebeen generated automatically. The final animation and simulation effectswill determine the setting of the control points at this phase. Somecomputational geometry techniques for skeletonization can also generatecontrol points at higher levels of the skeletal tree, providing thevisual and timely requirement can be met. These skeletonizationtechniques can use simplified representations of objects, e.g.discontinuities between contexts and context level control points. Thedivision between the two phases, i.e. automatic and manual setting ofcontrol points can be inherently fuzzy and application dependent.

[0089] Then, a quadtree (octree for 3D) is used to represent the objectfor space efficiency. This operation can be processed concurrently withall the others in FIG. 3. The quadtree (octree) will be used in thesimulation of object interaction as seen in the following description.Since a quadtree (octree) is generated recursively and the terminatingconditions can be user controllable, subdivision of the quadtree canterminate when each terminal quadtree node contains only one terminalskeletal tree node, thus associating the quadtree (octree) node with theskeletal tree node.

[0090] Besides building a skeletal tree bottom-up as described above,there are some other embodiments. Another embodiment can be building askeletal structure top-down using a known or future skeletonizationmethod, then building hierarchical bounding volumes (spheres, boxes,etc.) around some points at intervals on the skeletal structure. Theskeleton points can thus be adaptively adjusted to the centers of thebounding volumes. An advantage of this top-down approach is that thesegmentation process can be bypassed. That is, the segmentation can beconsidered as a sufficient but not necessary procedure.

[0091] In another embodiment, instead of having two data structures(skeletal tree and quadtree), a Voronoi region can be built around andassociated with each control point The Voronoi regions can have ahierarchical structure the same as that of control points.

[0092] In another embodiment, if a parallel architecture (such as meshcomputer, pyramid computer, etc. all with multiple processors) isutilized, the data structures can be dramatically different from theones described above. It is known in the area of parallel computing thatthe data structure is typically determined by the machine model.

[0093] The skeletal structure described above can not only used ingeometrically modeling object interaction, but also in generatingmulti-resolution volume hierarchy for physical simulation, in whichdifferential equations are used to analyze the interaction of forces andobject characteristics. That is, physical and geometric simulation canbe integrated through the volumetric skeletal structure.

[0094] Based on the data structures which have been built to representobjects, concurrent computing can be exploited to achieve realistic andreal time simulation. There can be two types of threads. One type runsat a general level, which roughly tests if objects will collide. Theother type runs at a detailed level, which accurately and in time checksthe interactions of objects. A general level thread can spawn a detailedlevel thread whenever necessary.

[0095]FIG. 5 shows a simple example in which a scene is made up of twooval objects. If multiple objects are involved, the scheduling scheme ofmultiple objects can be built on the top of the present method. Object 1and 2 are individually modeled in their local coordinate system. Supposeskeletal trees have been set up for them respectively (as roughly shownin Object 1). In the course of their movement, their control points arequeried by a general level thread to check if there will be a collision.The general level thread can map variables in individual localcoordinate systems to a world coordinate system, and take account ofobject orientation, relative moving directions. A search can beinitialized from the root of a skeletal tree, towards those controlpoints that are closer to another object in space. If the general levelthread conservatively determines, that the contexts where one pair ofcontrol points (CP₁ in object 1 and CP′₁ in object 2 as in FIG. 5) aresituated, may collide at some time, then a new detailed level thread canbe spawned to carry out accurate detection, and pass the pair of controlpoints to this detailed level thread.

[0096] The hierarchical depth in the skeletal tree where a general levelthread spawns a detailed level thread can vary. Depending on the objectsand the task-sharing requirements, it can be a user defined controlpoint, or a context level control point, or a sub-dividion level controlpoint. For context level tree nodes, the triggering condition fordetailed level threads can be that, if the distance between the controlpoints is less than 1.5 times the sum of the context dimensions, whichhave been calculated during the segmentation process.

[0097] The detailed level thread can use either a passive or an activemethod to detect collision. In a passive method, for each skeletal treenode in skeletal subtrees where CP₁ and CP′₁ are situated, from itscoordinates, the quadtree (octree) node which contains the skeletal treenode can be found. The passive method can then translate the positionsof terminal quadtree nodes branched from the nodes just determined to aworld coordinate system and check if there is any overlay (collision).If the terminal nodes of quadtrees (octrees), which are being checked,have only been subdivided into larger dimensions (length, height andwidth), they can be further subdivided by the detail level thread at runtime. The detailed level thread can also take advantage of relativepositions of objects, distribution of control points in space, as wellas object moving directions, to focus on a limited number of nodes inobjects, thus limiting the cost of computing resources. In an activemethod, wave fronts resembling context surfaces can be pre-sent alongthe objects' movement directions, again in a world coordinate system,thus predicting contact point and time. The wave fronts can beconstructed as parametric surfaces (such as spline surfaces) ifavailable, or they can use the node comparison approach as in thepassive method. The prediction of a collision position and time can beuseful for some applications, e.g. cutting of a graphic object. Theprediction can allow time for skeletal tree reorganization, which mayenable a timely generation of proper visual effects, e.g. an internalstructure at a proper location after cutting.

[0098] For collision response, the skeletal tree can be convenientlyused in the analytical simulation and the animation afterwards. Aftercalculating for deformation, the coordinates associated with therelevant nodes in the skeletal tree will be updated. Then, the skeletaltree can be used to guide the update of the quadtree (octree)representing objects. For example, a neighborhood of a control point,such as a tree node of the quadtree (octree), will move with the controlpoint. And the movement of nearby elements, such as other tree nodes ofthe quadtree (octree), can be interpolated. If an object is properlysegmented and subdivided, and the nodes in the skeletal tree representthe object in sufficient details, the updated quadtree (octree) willfaithfully represent the object after the update. For analytical andphysical simulation, with the assistance of predicting contactlocations, the skeletal tree of objects can be dynamically subdivided atthe prospective contact locations. That is, in run time, skeletal treeare further generated in greater details and thus volume objects aresubdivided into high-resolutions at the prospective contact locations.At the prospective contact locations, analytical and/or physicalstimulation, such as FEM (Finite Element Models) or the linked volumemodel, can also be conducted to stimulate deformation, possibly at somekey points and deformation of nearby elements can be interpolated.Therefore, in some sense, collision detection and response can beintegrated into one process. The procedure described above representsthe scheme in FIG. 1(a). Alternatively, the objects can be reprocessedagain after the collision occurs, as in FIG. 1(b).

[0099] Event driven methods can be used in general level and detailedlevel threads to speed up processing. For example, if one of the objectschanges movement direction, this event can generate messages to threads.The detailed level thread will probably not detect a collision any moreand terminate if a certain conservative condition is satisfied; thegeneral level thread may spawn another detailed level thread, and passanother pair of control points. The terminating condition fordetailed-level threads can be a minimum distance to that of thetriggering condition, or a number odf consective steps for the objectsto leave each other.

[0100] The usage of multithreads in the embodiments of the presentinvention is to increase responsiveness and efficiently use systemresources. Since for a graphical application, its main tasks may bephysical simulation and/or graphical rendering, multithreads can achieveinteraction modeling without conflicting with the main tasks. Anotherbenefit of responsiveness is detecting multiple contacts betweenobjects. It is possible not to use multithreads for some applications.That is, the principle described in this section can also be used insingle thread processing.

[0101] The principle discussed above can be used to detect multiplecontacts between objects, as well as the self-contact of one object.Compared with the previously proposed occupancy map methods, the presentmethod can represent objects economically and detect interactionsaccurately, and only consume a small amount of memory at run time.

[0102]FIG. 6 shows a class diagram of an embodiment to implementmodeling interaction of objects in UML (Unified Modeling Language)semantics. The use of Java Language is assumed in this class diagram.The members and methods of classes in FIGS. 7˜11 are only those onessignificant for the implementation.

[0103] The class graphobj in FIG. 6 simulates a graphical object and isshown in detail in FIG. 7. The method grabTask( ) opens an image filerepresenting an object and grabs image pixels into an array pixels. Themethod contourSet( ) tracks the edge of the object and saves contourpixels into a linked list pxllist. The methods localSegment( ) andglobalSegment( ) segment the object into a number of contexts atdiscontinuities according to local and global criteria respectively. Thelocal criteria can include local curvature and changes in slope, as wellas distances to neighboring discontinuities, while the global criteriacan include a global context dimension. The method dimensionEstimate( )estimates the dimensions of contexts and saves the results into an arraydimensions. The method subdivision( ) further subdivides contexts andsaves the subdivision locations as the indices to pxllist to anotherlinked list contextlist. The method contextSubtree( ) determines controlpoints of different contexts and their subdivisions using rule-basedfuzzy logic, and saves the calculated coordinates of control points intoa linked list contolptlist, in which each element holds the coordinatesof control point(s) for one context. Ideally, any calculated controlpoint initially lying outside the object should be mirrored inside. Themethod subtreeBuild( ) constructs a subtree for each context, and savesthe subtrees into a linked list subtreelist. For each context, everycontrol point is a treenode of its subtree and the coordinate of thecontrol point is associated with the treenode (using reference in Javalanguage). For a 2D object, the class Quadtree simulates the datastructure of a quadtree and can be an inner class of graphobj, similarlyfor 3D. The method quadsubdivide( ) recursively subdivides a object intoa QuadTree object qtree. The method quadsearch( ) returns for anarbitrary pixel within the object the smallest quadtree (octree) nodecontaining the pixel by recursively search down a QuadTree (Octree)object.

[0104] The class trajectory in FIG. 6 simulates a trajectory of anobject and is shown in detail in FIG. 8. The members of currentx andcurrenty represent the centerid of an object objxd of class graphobj ina global coordinate system a and b represent the increase along x and ydirections when objxd is stepped. The stepping happens when a classmethod stepping( ) of trajectory is called. The trajectory (representedby a and b) of an object can be set and randomized using methodssetTrajct( ) and randomTrajct( ) respectively. The current position ofan object can be also set and retrieved using methodssetCurrentPosition( ) and getCurrentPosition( ). The flag collidedindicates if objxd is collided with another object. This flag can be setand retrieved by methods setCollide( ) and isCollided( ) respectively.Trajectory can extend the class Thread in Java language. Generally,class trajectory can act as a bridge to transform points from localcoordinate system of objxd to a global coordinate system. It can alsoinclude methods that simulate translation, rotation and scaling of anobject, as well as relevant members and methods to transform betweencoordinate systems.

[0105] The class generalthread in FIG. 6 roughly tests if objects willcollide and is shown in detail in FIG. 9. Class generalthread extendsthe Thread class and overrides the runO method in Thread. Member trajctsis an array and each of its elements references a trajectory object. Themethod addTrajct( ) adds a number of trajectory objects to trajcts. Inthe simplest case, two objects of class trajectory will involve in aninteraction. The method moveExpect( ) compares the centreid positions ofobjects at previous simulation step with those at current stepmoveExpect( ) can return a boolean value indicating whether or notobjects are moved or an array representing distances between pairs ofobjects, which can be used for high level scheduling. If the high levelscheduling decides to further check a pair of objects, run( ) will callconservativeTest( ) method. The method conservativeTest( ) starts asearch from the roots of skeletal trees, whose coordinates can be thecenteroids of objects, towards those tree nodes closer to anotherobject. The searching can terminate at context level tree nodes, oralternatively can also terminate at tree nodes either higher or lowerthan the ones at context levels. Distances between these tree nodes ofdifferent objects are calculated, and can be compared with somecriteria. For context level tree nodes, a criterion can be that if thedistance is bigger than 1.5 times the sum of the context dimensionscalculated during the segmentation process. If these criteria aresatisfied for a pair of tree nodes, the method conservativeTest( )further tests if the pair is being checked by a detailthread object. Ifnot, the run( ) method will proceed to construct and initialize anobject of class detailthread for each pair of the tree nodes and callthe start( ) method of the newly created detailthread objects. Themethod collideLog( ) prints collision location(s) and simulation time,as well as saves pixels into an image file. An embodiment of run( )method of class generalthread is shown in the following pseudocode:While (true) { While ( moveExpect( ) != true ) { sleep for a while; } if( conservativeTest( ) == true )  // at least one detailthread object tospawn { for each close pair of tree nodes not in detailed checking {construct and initialize a detailthread object; call start( ) method ofthe detailthread object; } } sleep for a while; if iscollided( ) of anyelement in trajcts returns true { collideLog( ); break;  // this threadterminates } }

[0106] The class detailthread in FIG. 6 accurately checks interaction ofobjects and is shown in detail in FIG. 10. Class detailthread, whichextends the Thread class and overrides the run( ) method in Thread, hastwo trajectory members trajct1 and trajct2 which represent trajectoriesof two graphobj objects gpobj1 and gpobj2. The members of further andapartsteps indicate whether the two objects are leaving each other andthe consecutive steps of leaving apart, together they can indicateobjects will not collide and thus the present thread can terminate. Theconstructor detailthread( ) can take two trajectory objects and a pairof their tree nodes as arguments, and set trajct1 and trajct2 toreference the trajectory objects. The method nodedimSet( ) converts thecoordinates of control points to quadtree nodes by calling quadSearch( )method of gpobj1 and gpobj2, thus obtain the smallest quadtree nodescontaining the control points. This method can also check if thequadtree (octree) nodes for either object are identical if there aremultiple nodes involved. The method passiveCmp( ) converts localcoordinates of terminal quadtree (octree) nodes to a global coordinatesystem and check if the tree nodes of objects overlay in both x and ydirections for 2D (overlay in x, y, z directions for 3D). If overlayhappens, then collision is detected. The detection of collision cantrigger collision response processing, as well as setting flags,stopping ticking of movetimer object if so desired by applications. Themethod setCollide( ) calls setCollide( ) methods of trajct1 and trajct2.An embodiment of run( ) method of detailthread is shown in the followingpseudocode: While (true) { if( passiveCmp( ) == true ) // collided {setCollide( ); break; //this thread terminates } if( trajct1.isCollided()||trajct2.isCollided( ) ) // collision detected by other threads break;//this thread terminates if terminating conditions satisfied // objectshave been leaving apart for a while { call class method of space toupdate its class member checkingpair break; // this thread terminates }sleep for a while; }

[0107] The class space in FIG. 6 manages other classes and is shown indetail in FIG. 11, space has a class member checkingpair which holdsindices of skeletal tree nodes currently being checked in pairs bydetailthread objects. checkingpair can be implemented as a linked list.Before an object gthread of class generalthread spawns a newdetailthread object, elements of checkingpair are first checked. Whengthread spawns a new detailthread object or a detailthread objectterminates itself, the contents of checkingpair are updated by callingrelevant class methods, which read or write checkingpair. The methodsthat modify checkingpair should be synchronized. The methodinitializeAll( ) constructs and initializes objects of classes graphobj,trajectory and generalthread. Then in method startAll( ), the start( )methods of the thread objects are called. The method endTest( ) can testif the generalthread object gthread is alive for some applications.

[0108] Two other classes in FIG. 6 are movetimer and display. The classmovetimer simulates a clock ticker which calls the class methodstepping( ) of class trajectory at a fixed time interval; the classdisplay can interactively display the simulation of object interactionon a computer screen. Both can be implemented by those skilled in theart.

[0109] The present invention presents a new paradigm in modeling objectinteractions on the basis of volumetric representation and analysis, incontrast with previously proposed surface processing method. There canbe various alternatives, modifications, and equivalents, as well ashybrid models by combining the principle of the present invention withprevious modeling methods.

[0110] In the embodiment of the invention as seen in FIGS. 12 to 17, apth-degree NURBS (Non-Uniform Rational B-Spline) curve is defined by$\begin{matrix}\begin{matrix}{{C(u)} = \frac{\sum\limits_{i = 0}^{n}{{N_{i,p}(u)}\omega_{i}P_{i}}}{\sum\limits_{i = 0}^{n}{{N_{i,p}(u)}\omega_{i}}}} & \quad & {a \leq u \leq b}\end{matrix} & (1)\end{matrix}$

[0111] where the {P_(i)} are the control points (forming a controlpolygon), the {ω_(i)} are the weights, and the {N_(i,p)(u)} are thepth-degree B-spline basis functions defined recursively as$\begin{matrix}\begin{matrix}{{N_{i,0}(u)} = \left\{ \begin{matrix}1 & {if} & {u_{i} \leq u \leq u_{i + 1}} \\0 & {otherwise} & \quad\end{matrix} \right.} \\{{N_{i,p}(u)} = {{\frac{u - u_{i}}{u_{i + p} - u_{i}}{N_{i,{p - 1}}(u)}} + {\frac{u_{i + p + 1} - u}{u_{i + p + 1} - u_{i + 1}}{N_{{i + 1},{p - 1}}(u)}}}}\end{matrix} & (2)\end{matrix}$

[0112] where {u_(i)} are the knots which form a knot vector${U = {\underset{\underset{p + 1}{1\quad 2\quad 3}\quad}{\left\{ {a,\quad \ldots \quad,a,} \right.}u_{p + 1}}},\quad \ldots \quad,{u_{m - p - 1}\underset{\underset{p + 1}{1\quad 2\quad 3}\quad}{\left. {,b,\quad \ldots \quad,b} \right\}}}$

[0113] It is normally assumed that a=0, b=1, and ω_(i)>0 for all i.

[0114] Some of the important properties of the NURBS curves are

[0115] [1] Localness: Outside the interval [u_(i),u_(i+p+)1) the controlpoint P_(i) has no effect. The control points of the NURBS curve aresaid to exert the property of localness. If the control point P_(i) ismoved, or the weight ω_(i) is changed, it affects only that portion ofthe curve on the interval uε[u_(i),u_(i+p+)1).

[0116] [2] Convex hull property: if uε[u_(i),u_(i+1)), then C(u) lieswithin the convex hull of the control points P_(i−p), . . . P_(i). Thecurve is said to lie in the convex hull of the control polygon.

[0117] [3] Affine invariance: applying an affine transformation to thecontrol points is equivalent to applying an affine transformation to theNURBS curve; NURBS curves are also invariant under perspectiveprojections.

[0118] [4] Continuity: C(u) is at least p−k times differentiable where kis the multiplicity of a knot.

[0119] [5] Predictability: Let {overscore (u)} be a fixed parametervalue satisfying {overscore (u)}ε[u_(k),u_(k+p+)1), the effect ofmodifying the value of the weight ω_(k) is to move the pointC({overscore (u)}) along a straight line defined by P_(k) and theoriginal point C({overscore (u)}). As ω_(k) increases, the curve ispulled towards P_(k); as ω_(k) decreases, the curve is pushed away fromP_(k). Thus, weight modification has a perspective effect. In contrast,control point repositioning has a translational effect on a curve.

[0120] The present invention presents a new paradigm in modeling objectinteractions on the basis of volumetric representation and analysis, incontrast with previously proposed surface processing method. There canbe various alternatives, modifications, and equivalents, as well ashybrid models by combining the principle of the present invention withprevious modeling methods.

[0121] In relation to the facial muscles and their simulation using NURBcurves, it is seen that facial muscles are divided into four musclegroups: the jaw muscles, mouth muscles, eye muscles and row/neckmuscles. Facial muscles do not move independently, but affect eachother. Three types of muscle can be discerned as the primary motionmuscles:

[0122] [1] Linear muscles that pull in a linear direction, such as thezygomaticus major and corrugator.

[0123] [2] Sphincter muscles that squeeze, such as the orbicularis oris.

[0124] [3] Sheet muscles that behave as a series of linear musclesspread over an area, such as the frontalis.

[0125] Based on the properties of NURBS curves summarized, the NURBScurves can be so constructed that the visual effects of repositioningcontrol points and/or changing weights of control points can closelyapproximate muscle movements, i.e. the NURBS curves can geometricallysimulate the anatomical characteristics of facial muscles. A NURBS curvecan be equivalent to an individual muscle or a group of muscles.Furthermore, changing the weights of different control points cansimulate the movement of linear, sphincter, or sheet muscles. Forexample, a linear muscle can be simulated by changing the weight of onecontrol point; while a sphincter or sheet muscle can be simulated byupdating the weights of several neighboring control points. Duringanimation, an individual vertex on the facial model can be affected byone or more NURBS curves, which is intuitively analogical to real facialmovements.

[0126] Human face modeling and animation in the present method can bedivided into three consecutive procedures: build NURBS curves tocorrespond with facial anatomy; associate vertices with NURBS curves;animate facial expressions. The first two procedures together can beconsidered as initialization.

[0127] Cubic curves are used in face modeling since they offer shapeflexibility and also avoid unwanted oscillations. Chord lengthparametrization is used to determine knots of the NURBS curves.

[0128] The positions of the control points are primarily decided by thelocations where the simulated muscles are attached to facial bones. FIG.12 shows the general positions of the control polygons of the NURBScurves, and Table 1 shows their simulated muscles. TABLE 1 NURBS curvesin FIG. 12 and their simulated muscles Curve Index Simulated Muscles 1,2 frontalis, corrugator 3, 4 orbicularis oculi 5 zygomaticus major,levator labii superioris, triangularis, depressor anguli, depressorlabii inferioris, mentalis 6 orbicularis oris

[0129] To realistically simulate the pulling effect of muscles, the 3Dpositions of control points are shifted slightly above the skin surfaceas in FIG. 13. Each linear muscle can be simulated by positioning acontrol point near the location where the muscle attaches to a bone. Theclosed NURBS curves (curves 3, 4 & 6 in FIG. 12) are used to simulatethe sphincter muscles around the eyes and mouth. The muscles on the twosides of the cheek are separated by the nose, so the two ends of theNURBS curve simulating these muscles (curve 5 in FIG. 12) are alsoseparated by the nose in the simulation model.

[0130] Due to the localness of the NURBS curve, a muscle can also besimulated by several control points, or vice versa, to generate morerealistic or artistic effects.

[0131] According to the movements of human facial muscles, vertices canbe separated into two top-level units, each with two sub-units (asroughly shown in FIG. 14):

[0132] 1) Vertices where muscles affect the eyes. They are furtherdivided into

[0133] a) Vertices on the forehead;

[0134] b) Vertices on the eyes.

[0135] 2) Vertices where muscles affect the mouth. They are furtherdivided into

[0136] a) Vertices on the cheeks and jaw;

[0137] b) Vertices on the mouth.

[0138] To more realistically simulate the mutual influence of facialmuscles and their gradual transition, fuzzy sets can be used to assignany vertex a membership to a facial unit. The sub-units within the twotop-level units can be considered as fuzzy sets; while the two top-levelunits themselves can be considered as classical crisp sets, since thereis little interaction between the muscles on these two areas. Thecontrol polygons of the NURBS curves can also act as boundaries betweenunits.

[0139] Thus each vertex on the facial model will be first decided asbelonging to one of the top-level units, which can be done by comparingthe vertex's coordinates with the unit boundary. Then, membership to thesub-units will be determined for this vertex. In the implementation,calculation of membership is integrated into connection of the vertexwith sample points of NURBS curves.

[0140] For any vertex Vn, a gradient Ln passing through it will first becalculated before determination of membership calculations.

[0141] The distribution of muscles around the mouth is more complex thanaround the eyes. The orbicularis oris, which circles the mouth, is thelocation of insertion for many other muscles. Some of these musclesinsert into the corner of the mouth, while others insert into the upperand lower lips. FIG. 15 shows the determination of gradient Ln for avertex around the mouth. As shown in FIG. 15, V1 and V2 are two verticesaround the mouth, vect1 to vect4 are preset vectors resembling themuscles, kpt1 and kpt2 are two key points representing a corner and thecenter of the mouth, respectively. First, connect the vertex (V1 or Y2)and kpt1, the 2D angle (on the x-y plane) between this connection andthe positive x-axis is calculated. If this angle is between that ofvect1 and vect2, then this connection remains, as L1. Otherwise, a newconnection L2 will be determined, with a weighted gradient in 3D spaceas $\begin{matrix}{{{\frac{d3}{{d1} + {d3}}{gd1}} + {\frac{d1}{{d1} + {d3}}{gd3}}},} & (2)\end{matrix}$

[0142] where d1 and d3 are distances of V2 to vect1 and vect3, gd1 andgd3 are 3D gradients of vect1 and vect3, respectively. Point O is thepoint on L2 with the shortest distance to the line section between kpt1and kpt2.

[0143] The line Ln passing Vn can be vertical for vertices affected bycurve 3, 4 and 6 which simulate sphincter muscles, and for thoseaffected by curve 1 and 2 which simulate sheet muscles. Intuitively, thevertices on the upper part of the eyes or the mouth will be connectedwith sample points on the upper section of the closed NURBS curve, viceversa for the vertices on the lower part. For the vertices affected bythe control points (of curve 1 and 2) which simulate the corrugator,i.e. the frown muscle, the gradient of line Ln can be slightly downwardsto the medial end horizontally, just as the contracting direction of thecorrugator.

[0144] The association of the NURBS curves (as in FIG. 12) with thevertices in the facial units (as in FIG. 14) is shown in Table 2. Theassociation is determined by calculated membership values. TABLE 2Association of vertices with NURBS curves Unit No. Facial Area SideIndex of Associated Curve(s) 1 forehead left 1 right 2 2 upper eyelidsleft 1, 3 right 2, 4 2 lower eyelids left 3 right 4 3 cheek & jaw 5 4mouth 5, 6

[0145] Each vertex is connected with sample points of its associatedcurves, each of which has some sample points at intervals of μ. As shownin FIG. 16, line Ln passes through point Vn, whose gradient has beendetermined. By calculating the distance between any relevant samplepoint Si of the NURBS curve and the line Ln in 3D space, Vn can beconnected to the Si via the shortest distance.

[0146]FIG. 17 shows an example of determining membership to forehead andone eye for a vertex on the area. Since membership calculation can beindividually processed for each vertex, D1 and D2 in FIG. 17 are set asrelative values.

[0147] Each vertex is associated with a ratio of the movement to itsconnected sample point on a curve. The ratio r of Vn can be determinedas: $\begin{matrix}{{r = {K \cdot {\sin \left( {\mu \cdot \frac{\pi}{2}} \right)}}},} & (3)\end{matrix}$

[0148] where μ is the membership (0≦μ≦1) of Vn; k is a constant with theeffect of a spring constant. The sine function is used to smooth andconstrain the movement of vertices.

[0149] During initialization, the vertices are associated with the NURBScurves while all NURBS curves are set their weights to 1. Then, based onanatomical knowledge, arrays are set to the weights of the maximumexpressions. To generate an expression, movements of the curves arelinear interpolated. It is possible to modify the curves directly byusing weights as variables, but the drawback of this approach is thatthe weights need to be changed exponentially in order to generate linearcurve movements.

[0150] In another embodiment, a vertex is connected with a curve samplepoint based on the minimum distance. Each vertex is associated with aratio of movement of its closet sample point on a curve. The calculationof the ratio is shown in FIG. 18. As shown in this FIG. 18, X, A1 and 42approximately represent the positions where the muscles are attached tothe skull, while B1 and B2 roughly represent the positions where themuscles are embedded into the soft tissue of the skin. A1 and A2 arealso control points of the NURBS curve. V is a vertex and connected to acloset sample point S on a NURBES curve. E and F are the projections ofVon A1A2 and B1B2 respectively. G is the projection of V on line sectionEF. All the calculations are conducted in 3D space. The ratio r of V tothe movement of S is determined by the following pseudocode:

[0151] if G is at the outer side of A1A2,

[0152] r=0

[0153] else if G is between E and F$\quad {r = {k_{1}\quad {\sin \left( \frac{\pi {{GE}}}{{{2}{FE}}} \right)}}}$${{else}\quad {if}\quad 0} \leq {f_{x}\frac{{GF}}{{FE}}} \prec \frac{\pi}{2}$$\quad {r = {k_{2}\quad {\cos \left( {f_{x}\frac{{GF}}{{FE}}} \right)}}}$

[0154] else r=0

[0155] here k₁ and k₂ have the equivalent effect of spring constants;f_(x) is a frequency coeficient anatomically representing a muscleeffected area. The sine and cosine functions are used to constrain andsmooth movement of vertices.

[0156] During animation, changing the weights will modify the NURBScurves. Then the updated position of a vertex V is $\begin{matrix}{{\overset{\rightarrow}{{VV}^{t}} = {\sum\limits_{i = 1}^{t}{r_{i}\overset{\rightarrow}{{SiSi}^{\prime}}}}},} & (4)\end{matrix}$

[0157] where V′ is updated position of V; Si′ is the new position of asample point Si which is an associated sample point of V; t is the totalnumber of NURBS curves affecting V; and r_(i) is a calculated ratio.Thus, any vertex on the facial model can be controlled by a number ofNURBS curves.

[0158] For example, for the expression of happiness, the most prominentmuscle action is the contracting of zygomaticus major, which is locatedaround the upper and lower lips at one end, and attaches just before theear at the other end. The effect of this muscle action is raising themouth upward and outward. This action is simulated by increasing theweights of control points which are also located near the ear, for NURBScurve 5 in FIG. 12. A secondary feature for happiness is the slightcompressing of eyelids, which is simulated by decreasing the weights ofseveral upper control points of NURBS curves 3 and 4 respectively.

[0159] Since a NURBS curve can represent a group of muscles, changingdifferent weights can simulate movement of the different muscles. Forexample, the NURBS curve on one side of the forehead has several controlpoints (curve 1 or 2 in FIG. 12). Increasing the weights of the middlecontrol points can simulate the contracting of the frontalis, i.e. thebrow lifting muscle; increasing the weight of the control point roughlysituated between the eyes can simulate the contracting of thecorrugator, i.e. the frown muscle.

[0160] The effect of skin wrinkles can be simulated simultaneously asthe pulling effect of muscles. While calculating the scalar ratio r, avector associated with the vertex V can be also calculated to simulatewrinkles. The direction of the vector can be perpendicular to themovement of the associated sample point S, the magnitude of the vectorcan be determined as $\begin{matrix}\left\{ {\begin{matrix}{0,} & {{{if}\quad h_{w}} = {{k_{w}\quad {\sin \left( F_{w} \right)}} < h_{t}}} \\{h_{w},} & {{{if}\quad h_{w}} = {{k_{w}\quad {\sin \left( F_{w} \right)}}>=h_{t}}}\end{matrix}\begin{matrix}\quad & {F_{w} = {f_{w}R}} \\\quad & {\quad {or}} \\{where} & {F_{w} = {{f_{w}R} - {H_{r}*{random\_ generator}{()}}}}\end{matrix}} \right. & (5)\end{matrix}$

[0161] where h_(t) acts as a threshold and R=|{right arrow over(Ob_(n))}| as in FIG. 5(b); k_(w) acts as a spring constant and biggercan produce deeper wrinkles; f_(w) is a frequency coefficient and biggerf_(w) can produce a larger number of wrinkles. To more realisticallysimulate wrinkles, the function random_generator( ) generates a randomnumber between 0.0 and 1.0 (as the method Math.random( ) in JavaProgramming Language); H_(r) is a natural number such as 1 or 2, andshould satisfy H_(r)<2π.

[0162] In another embodiment, instead of changing 3D coordinates ofvertices, the normal of the polygons can be changed so the areas appearto be darker after rendering.

[0163] The NURBS curve belongs to the large family of parametricrepresentations. The principle disclosed in this specification may beimplemented by using another representation.

[0164] The present invention use polygon representation to model anddeform human faces. The same principle can be used in other objectrepresentations, such as parametric patch nets; it can also be used tosimulate other surface deformations, such as animal facial movements ormovements of other human organs.

[0165] The present invention simulates the facial muscles by changingthe weight of the NURBS curves. As a NURBS curve is defined by itscontrol points, weights and knots, modifying any of these changes theshape of the curve. Furthermore, since a NURBS curve can be presenteddifferently using reparameterization, it is possible for one skilled inspline theory and application to come up with endless solutions tochange a curve shape using means other than weight modification.

[0166] The mouth can generate enormous expressions in real life. Anotherembodiment is to allow the modification of curve 5, which simulates themuscles around the mouth as in FIG. 12, to affect the shape of curve 6,which simulates the sphincter muscle around the mouth, and theassociation between curve 6 and vertices representing the mouth.

[0167] Another embodiment is to associate the mouth vertices withmultiple NURBS curves for different expressions, instead of two curves(curve 5 and 6 in FIG. 12), and turn on/off the effect of these curvesdepending on the desired expression.

[0168] Another embodiment is to employ multiple NURBS curves on a facialunit, especially those units with big surface areas, such as the cheekand forehead. When the density of the curves and their sample pointsreaches the requirement of the Sampling Theorem, the procedure ofassociating vertices with NURBS curves can be greatly simplified. Thus,any vertex can be associated with a sample point of a NURBS curve basedon the minimum distance, and a ratio of movement may not be needed.Modifying the NURBS curves and thus moving the vertices can simulatefacial expressions.

[0169] Modeling facial movements using NURBS curves is a geometricalmodeling method. The NURBS curves are used to control and coordinate themovement of the vertices. This approach can be combined with physicallybased models, which usually means differential equations and/orbiomechanical models are used to analyze the interaction of forces,object characteristics and deformations. For example, a muscle'sforcefulness can be affected by its length relative to the restposition, and movement of other muscles. So physically based method canbe used to update the coefficient k in formula (3).

[0170] The muscle movement can also be more accurately simulated, byapplying physically based model and vector analysis. For example, as inFIG. 19, suppose S is a sample point on a NURNS curve, and {right arrowover (SS′)} is the vector represents the movement of S due to weightchanging. Suppose an anatomically based and physically modeled muscle isplaced near {right arrow over (SS′)}. Then {right arrow over (SS′)} isvector decomposed with one vector along the muscle as {right arrow over(PQ)}. After physically based calculation, a more realistic movement of{right arrow over (PQ)} can replace {right arrow over (SS′)} in formula(4).

[0171] Using of NURBS curves can also achieve both realisticallyphysical modeling and real-time performance. For example, deformationsof a few key points on the facial model can be calculated usingphysically based modeling, then NURBS curves can be used tointerpolate/coordinate movements of the vertices among and around thesekey points.

[0172] As a coding, normal calculation for polygon representation hasbeen appeared in numerous literatures.

[0173] For NURBS curves, there are algorithms to generate basisfunctions and NURBS curves. Alternatively, the following (Visual C++)code can be used to generate a section of cubic NURBS curve: int f1 =pskv − p; // pskv — current position in knot vector; p — degree of NURBScurve (3 for cubic) int s2 = f1 + 1; int t3 = s2 + 1; int f4 = t3 + 1;double numerator_x, numerator_y, numerator_z, denominator, x, y, z;double sec1, sec2, sec3, sec4; // array v[] holds the knot vector, arrayweight[] holds the weights of control points while (u < v[pskv+1]) {sec1 = weight[f1]* pow(v[s2+p]−u,3.)/((v[s2+p]−v[s2])*(v[s2+p]−v[t3])*(v[s2+p]−v[f4])); sec2 =weight[s2]*((u − v[s2])* pow(v[s2+p] − u,2.)/((v[s2+p]−v[s2])*(v[s2+p]−v[t3])*(v[s2+p]−v[f4])) + (v[t3+p]−u)/(v[t3+p] − v[t3])*((u − v[t3])*(v[s2+p] − u)/((v[s2+p] −v[t3])*(v[s2+p] − v[f4])) + (v[t3+p] − u)*(u − v[f4])/((v[t3+p] −v[f4])*(v[s2+p] − v[f4])))); sec3 = weight[t3]*((u − v[t3])/((v[t3+p] −v[t3])*(v[s2+p] − v[f4]))*((u −v[t3])*(v[s2+p] −u)/(v[s2+p]−v[t3]) +(v[t3+p] − u)*(u − v[f4])/(v[t3+p] − v[f4])) + (v[f4+p] − u)* pow(u −v[f4], 2.)/((v[f4+p] − v[f4])*(v[t3+p] − v[f4])*(v[s2+p] − v[f4])));sec4 = weight[f4]* pow(u − v[f4], 3.)/((v[f4+p] − v[f4])*(v[t3+p] −v[f4])*(v[s2+p] − v[f4])); denominator = sec1 + sec2 + sec3 + sec4; if(denominator == 0.0) { cout << “Divided by zero.\n” >> end1; exit (−1); }// array ctrpt[] holds the coordinates of control points of the NURBScurve, // each element in ctrpt[] is a point object numerator_x =ctrpt[f1].x*sec1 +ctrpt[s2].x*sec2 + ctrpt[t3].x*sec3 +ctrpt[f4].x*sec4; x = numerator_x /denominator ; numerator_y =ctrpt[f1].y*sec1 +ctrpt[s2].y*sec2 + ctrpt[t3].y*sec3 +ctrpt[f4].y*sec4; y = numerator_y /denominator ; numerator_z =ctrpt[f1].z*sec1 +ctrpt[s2].z*sec2 + ctrpt[t3].z*sec3 +ctrpt[f4].z*sec4; z = numerator_z /denominator ; // array curvept[]holds the sample points of the NURBS curve // constructor of pointobject takes three coordinates curvept[cpcnt] = point (x, y, z); u +=intv; // increase u a fixed interval

[0174] The foregoing describes only some embodiments of the presentinventions, and modifications obvious to those skilled in the art can bemade thereto without departing from the scope of the present invention.

1. A method of context based skeletal tree generation for use in animation and simulation of object interactions, said method comprising the steps of preprocessing graphic objects, segmenting processed graphic objects into a plurality of contexts at surface slope and curvature discontinuities, generating control points to represent said contexts, building a volumetric skeletal representation from said control points, and wherein said volumetric skeletal representation is manipulated to facilitate animation and simulation of object interactions.
 2. The method according to claim 1, wherein the volumetric skeletal representation is in one form a skeletal tree and the means to generate the skeletal tree includes both bottom-up and top-down.
 3. The method according to claims 1 or 2, wherein rule based fuzzy logic is used to generate said control points.
 4. The method according to cany one of claims 1 to 3, wherein the step of segmenting preprocessed graphic objects into a plurality of contexts, includes edge detection, edge tracking and curvature discontinuity detection.
 5. The method according to claim 4, wherein said edge detection is accomplished by using gradient or Laplacian operators, said edge tracking by using omnidirectional tracking, and curvature discontinuity detection is accomplished by measuring the k-slope and k-curvature and comparing them to predetermined threshold values.
 6. The method according to claim 5, wherein the threshold values and kare Application dependent, the value of k being set to 5 to 10 for images of various sizes.
 7. The method according to claim 6, wherein the absolute value for threshold of k-curvature (C¹) is set to 10 degrees; the absolute value for threshold of the change in k-slope (C⁰) between neighboring pixels being set to 20 degrees.
 8. The method according to claim 5, including the steps of providing data structures representing contexts of the segmenting process, said data structures having a queue containing x-y co-ordinates and/or chain code values of context points and pointers to neighbouring contexts.
 9. The method according to claim 6, including the steps of applying context dimension estimation and/or resegmentation.
 10. The method according to claim 7, wherein the step of applying context dimension estimation is accomplished by scanning the context both horizontally and vertically, constructing a run-length histogram which gives the occurrence of length of different runs within the context, the dimension of the context being the run-length corresponding to the maximum value of the histogram.
 11. The method according to claim 10, further including the steps of comparing dimensions of different contexts and selecting a standard dimension, and using the standard dimension as aparameter to re-segment.
 12. The method according to claim 11, wherein the step of applying fuzzy logic includes the step of providing a quadtree or octree to represent the object for space efficiency.
 13. The method according to any one of the preceeding claims wherein there are two types of threads, one type runs at a general level, which roughly tests if objects will collide, the other type runs at a detailed level, which accurately and in time checks the interactions of objects and a general level thread can spawn a detailed level thread whenever necessary.
 14. The method according to claim 13, wherein if multiple objects are involved, the scheduling scheme of multiple objects can be built on the top, wherein the objects are individually modeled in their local coordinate system and skeletal trees are set up for them respectively, and in the course of their movement, their control points are queried by a general level thread to check if there will be a collision.
 15. The method according to claim 14, wherein the general level thread maps variables in individual local coordinate systems to a world coordinate system, and take account of object orientation, relative moving directions, a search is initialized from the root of a skeletal tree, towards those control points that are closer to another object in space, and if the general level thread conservatively determines, that the contexts where one pair of control points are situated, may collide at some time, then a new detailed level thread can be spawned to carry out accurate detection, and pass the pair of control points to this detailed level thread.
 16. The method according to claim 15, wherein the hierarchical depth in the skeletal tree where a general level thread spawns a detailed level thread can vary, and depending on the objects and the task-sharing requirements, it can be a user defined control point, or a context level control point, or a sub-division level control point.
 17. The method according to claim 16, wherein for context level tree nodes, the triggering condition for detailed level threads can be that, if the distance between the control points is less than 1.5 times the sum of the context dimensions, which have been calculated during the segmentation process.
 18. The method according to claim 17 wherein the detailed level thread can use either a passive or an active method to detect collision, in a passive method, for each skeletal tree node in skeletal subtrees where control points are situated, from its coordinates, the quadtree or octree node which contains the skeletal tree node can be found whereby the passive method then translates the positions of terminal quadtree nodes branched from the nodes just determined to a world coordinate system and check if there is any overlay or collision, if the terminal nodes of quadtrees or octrees, which are being checked, have only been subdivided into larger dimensions (length, height and width), they can be further subdivided by the detail level thread at run time, wherein in an active method, wave fronts resembling context surfaces are pre-sent along the objects' movement directions, again in a world coordinate system, thus predicting contact point and time, the wave fronts are constructed as parametric surfaces if available, or they can use the node comparison approach as in the passive method.
 19. The method according to claim 18, wherein for collision response, the skeletal tree can be conveniently used in the analytical simulation and the animation afterwards, after calculating for deformation, the coordinates associated with the relevant nodes in the skeletal tree will be updated. Then, the skeletal tree can be used to guide the update of the quadtree (octree) representing objects, and the movement of nearby elements, such as other tree nodes of the quadtree (octree), can be interpolated. If an object is properly segmented and subdivided, and the nodes in the skeletal tree represent the object in sufficient details, the updated quadtree (octree) will faithfully represent the object after the update.
 20. The method according to claim 19, wherein the event driven methods are used in general level and detailed level threads to speed up processing.
 21. The method according to claim 20, wherein usage of multithreads is to increase responsiveness and efficiently use system resources for interaction modeling without conflicting with the main tasks.
 22. A method to use a volumetric skeletal representation representing an object for interaction detection and response, wherein nodes are searched for detection and position of nodes are updated for response, said representations being generated either bottom-up or top-down or both.
 23. The method according to claim 22, wherein the skeletal representations include a tree and he nodes includes treenodes.
 24. The method according to claim 23, wherein the method to generate the skeletal tree includes context-based approach, fuzzy rule approach, automation in processing.
 25. The method according to claim 24, wherein the use of trees in tree/concurrent computing in detecting interactions includes multithreads.
 26. The method according to claim 25, wherein collision locations and time are predicted and he skeletal trees are updated in interaction in response for continuous stimulation.
 27. The method according to claim 26, wherein the method includes an active method which dynamically generates the tree in greater detail in prospective locations of the collision.
 28. A method of using parametrical representation to control and represent surface deformation to simulate movement in animation, said method including the steps of building parametrical representations to represent vertices in the topography of the surface, associating said vertices with different ones of the parametrical representations to effect movement thereof, and animating movement of the vertices to simulate relative motion therebetween.
 29. The method according to claim 28, wherein said method is used to animate facial expression, whereby the vertices in the topography of the surface are assigned to various portions of facial anatomy including muscles.
 30. The method according to claim 29, wherein the parametrical representations are weighted to differentiate the different portions of the facial anatomy.
 31. The method according to claim 30, wherein weighting parameter is changed to simulate facial expressions.
 32. The method according to claims 28 to 31, wherein the parametrical representations are NURBS curves.
 33. The method according to claim 32, wherein position of the NURBS curves are predetermined by the location where simulated muscles are attached to facial bone, each NURBS curve having at least one control point.
 34. The method according to claim 33, wherein fuzzy logic is used to determine the association of vertices with NURBS curves.
 35. The method according to claim 34, wherein the weights of the NURBS curves are variable to modify the NURBS curves.
 36. The method according to any one of claims 28 to 35, wherein the parametrical represntations and/or their changes are sent to a remote location via a communication link to generate deformation at remote location.
 37. A method of using parametrical representation to control and represent surface deformation to simulate movement in animation, said method including the steps of building parametrical representations to represent vertices in the topography of the surface, associating said vertices with different ones of the parametrical representations to effect movement thereof, and animating movement of the vertices to simulate relative motion therebetween whereby quantitative simulation is achieved.
 38. The method according to claim 37, wherein the parametrical representation/vector analysis, parametrical representation/physical based methods. Parametrical representation/vector analysis/physical based methods are combined 